feat(KV-FP8): the CUDA fp8 KV store and read, reached by removing the W1 guard that refused CUDA before the provider table (#1593) - #1606

feat(KV-FP8): the CUDA fp8 KV store and read, reached by removing the W1 guard that refused CUDA before the provider table (#1593)

KV-FP8 W1 landed the CPU half in 2026-07 and left the CUDA arm as a named
later brick. It is now the critical path of benchmark campaign #1574, whose
subject r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121 declares
kv_cache_quant_algo: "FP8" in hf_quant_config.json, so no cell of that
three-way can be served correctly without it.

W1 is the ORACLE for this wave. Every gate here compares CUDA to the landed CPU
kernels. Nothing re-ports the numerics.

The store

An ELEMENTWISE-IDENTICAL port of the fp8 branch of vLLM's
reshape_and_cache_flash_kernel (csrc/libtorch_stable/cache_kernels.cu:314-401)
plus CopyWithScaleOp (:241-252) at pin 555967922, restricted to upstream's
is_contiguous_heads && kv_scale_stride == 0 arm (:352-366). That is the only
arm the op's wrapper admits, because the vt cache is the NHD unbind slice and
ReshapeAndCacheFp8 takes two scalar scales.

It is not instruction-identical, and calling it a 1:1 port would overstate it.
Upstream's contiguous-heads arm moves the row through
vectorize_with_alignment<VEC_SIZE> (:360-363, VEC_SIZE 8 for a 2-byte
source and 4 for f32); this one is a scalar strided loop over the same elements
in the same order. Same bytes out, fewer bytes per instruction. W4 owns the
bandwidth.

The converter is upstream's own
__nv_cvt_float_to_fp8(hp / scale, __NV_SATFINITE, __NV_E4M3)
(quant_utils.cuh:497-503). It is a true DIVIDE, not the activation path's
hoisted reciprocal multiply, and the two differ by up to one f32 ulp before the
round. Its byte-for-byte equality to the CPU software codec vt::F32ToF8E4M3 is
already MEASURED at zero tolerance on sm_110 and sm_121a
(.agents/specs/vt-fp8-quant-arch-gate.md G2).

The read

LoadKv(ptr, i, scale) joins Load. It is INERT on the f32 and bf16 arms, which
forward to Load unchanged, so every existing caller reads the same bytes in the
same order. On uint8_t it is upstream's scaled_vec_conversion<float, uint8_t>
(quant_utils.cuh:419-429), written as the SAME ARITHMETIC as vt::F8E4M3ToF32
so that CUDA equals CPU on the read as a property of the source rather than of a
measurement.

Same arithmetic holds for all 254 FINITE e4m3 codes and not for the two NaN ones.
On 0x7F and 0xFF the CPU returns std::numeric_limits<float>::quiet_NaN()
(0x7FC00000) and the device returns CUDART_NAN_F (0x7FFFFFFF): both quiet,
both propagating, different payload. No gate here can see it, because a NaN
compares unequal to itself, so a byte or NMSE comparison fails on any payload
rather than on the wrong one. Reaching it also needs a non-finite input, since
__NV_SATFINITE clamps an out-of-range magnitude to 0x7E or 0xFE.

Only the two correctness-grade kernels serve fp8, which is what the existing
ladder already implies. The WMMA prefill kernels stage bf16 fragments, the
vendored FA-2 launchers take bf16 pointers, and the vectorized decode-opt and GQA
kernels read through LoadRowN and LoadRow8, 128-bit uint4 loads specialized
for bf16 and f32 only. Upstream draws the same line from the other side, at
flash_attn.py:181-187,796-805. A tensor-core fp8 read is a performance brick;
W2's gate is parity and W4 owns the memory measurement.

What actually made the arm unreachable

Neither kernel. Both W1 wrappers carried
VT_CHECK(q.device.type == DeviceType::kCPU, ... "a named later brick")
evaluated BEFORE provider lookup, so no CUDA kernel could ever have been reached
however well it was registered. That is the RED this change was written against.

The STORE now resolves through the provider table like every other op, because
kReshapeAndCacheFp8 is its own OpId that only CPU and CUDA register, so an
unimplemented backend refuses by name inside GetOp. The READ cannot: it rides
ADDITIVE fields on PagedAttentionArgs of an op kMETAL and kROCM already
register for the FLOAT path, and nothing in the provider table separates the two
arms, so an fp8 cache would reach a float kernel and return silent garbage. It
keeps an explicit CPU-or-CUDA list whose message names the missing part.

Unreached, deliberately, and not new

Nothing calls the fp8 KV path from a production entry point on either backend.
vt::ReshapeAndCacheFp8 and PagedAttentionArgs::kv_cache_dtype have no caller
outside their tests. W1 landed in that state and this does not change it.
KV-FP8 W3 owns the wiring: half-sized KV blocks in the runner,
--kv-cache-dtype threaded from the CLI, and the checkpoint k_scale and
v_scale path. #1593 tracks it, and .agents/specs/fp8-kv-cache.md lists it
under ## Owed.

The CUDA translation units COMPILE; the device gates are UNEXECUTED

Those are two different states and the first version of this body ran them
together. CI job cuda-fat-build (.github/workflows/ci.yml:773) configures
-DVLLM_CPP_CUDA=ON with
-DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a', builds
target vllm, and reaches every CUDA translation unit with
-Werror=all-warnings. Both files this change touches are unconditional sources
of that target. Line anchors as read at the tree CI built,
4d71e776efc18cb5e61a26e642ddad8de5339134, and not at this branch's merge base:
cmake/CompilerWarnings.cmake:53, CMakeLists.txt:1694 and :1697. That job
PASSED on that same SHA: run 32495320287, job 96812232428.
So the CUDA arm was uncompiled by the IMPLEMENTING session, which had no nvcc
and no device, and has since been compiled for ten architectures.

Execution is still absent, because that same job configures
-DVLLM_CPP_BUILD_TESTS=OFF and therefore never links or runs
tests/vt/test_cuda_fp8_kv_cache.cpp. G2 (provider registration), G3 (store byte
parity over the f32, bf16 and f16 sources), G4 and G4b (paged-read parity over
the decode and prefill shapes, for the f32 and the bf16 query/output
instantiation) and G5 (the CUDA kernel's own e5m2 refusal) all skip with a
MESSAGE naming what did not run. G1 and G1b, provider routing and the
Metal/ROCm refusal, are the only cases that ran, and they run on the CPU leg.

A CPU-only gate cannot see a CUDA defect, and the fresh review PROVED this one
blind rather than arguing it: deleting the entire production call site for the
CUDA fp8 read left the focused gate 100% green, and inverting the CUDA store's
scale direction produced ninja: no work to do, because a CPU build's
compile_commands.json contains ZERO .cu translation units — 1046 entries in
the review's configure, 1030 in this session's, not one of them CUDA. That
is the honest consequence of the state above. The first rc lease that touches
this row must run ctest -R test_cuda_fp8_kv_cache before W2 counts as measured.
The spec's ## Owed says so, and the wave table's DONE means landed and gated
rather than measured on hardware.

What the fresh review changed

Six findings, all repaired here.

Four new sites plus this body cited scaled_vec_conversion<float, uint8_t> at
quant_utils.cuh:302-308. At the pin, 301-305 is the GENERIC primary template
that returns its input unchanged and 307-314 is the fp8-to-HALF
<uint16_t, uint8_t> specialization. The <float, uint8_t> one is at
:419-429. Fp8KVCacheDataType was cited at dtype_fp8.cuh:9-13, the
#include <cuda_fp8.h> guard; the enum is at :15-19. Three W1 copies of the
same wrong anchor are outside this change's authority and are filed as #1636.

G5 could not fail for any CUDA-side defect. It asserted a bare throw that
originates in the op wrapper, ABOVE the device checks and above GetOp. The
e5m2 refusal exists in THREE places — that wrapper, the CPU kernel and the CUDA
kernel — and both halves of the finding were measured rather than argued.
Deleting the CUDA kernel's VT_CHECK on a CPU build gives ninja: no work to do and leaves the file 7 cases / 10 assertions SUCCESS!. Deleting the
wrapper's leaves W1's test_ops_fp8_kv_cache GREEN at 8/511, because execution
falls through to the CPU kernel's own check and W1's refuses e5m2 case
(tests/vt/test_ops_fp8_kv_cache.cpp:342) asserts CHECK_THROWS_AS on
std::runtime_error rather than a message; only deleting the wrapper AND the
CPU kernel check turns it red, at 7/8 cases and 510/511 assertions. So what W1
pins is "e5m2 is refused somewhere on the CPU path", and a layered refusal needs
an assertion that NAMES its layer. G5 now resolves the registered provider with
GetOp(OpId::kReshapeAndCacheFp8, DeviceType::kCUDA), calls it directly, and
requires the message to carry both cuda reshape_and_cache_fp8 and fp8_e5m2,
which no other layer produces. Three mutations, each rebuilt, run and restored
against a pre-taken sha256.

LaunchPagedFp8Out<__nv_bfloat16, __nv_bfloat16> — the instantiation a served
bf16 model takes, and the one #1574's subject runs — was entirely ungated, and
the store's DType::kF16 arm was untouched while the wrapper's IsFloat()
admits it. G4b and the f16 leg of G3 close both.

Two prose overstatements are now true: the store is elementwise-identical rather
than 1:1, and the read is the CPU codec for every finite code rather than "line
for line".

Evidence

RED first, in the implementing session: test_cuda_fp8_kv_cache reported 6
assertions and 6 failed, on reshape_and_cache_fp8: only the CPU fp8-KV store is implemented in W1 at ops.cpp:3478 and paged_attention: only the CPU fp8-KV read is implemented in W1 at ops.cpp:3811.

Three negative mutations from that session, each rebuilt, run and restored
against a pre-taken sha256. Reinstating the store guard failed 3 assertions.
Reinstating the read guard failed 7. Deleting the Metal/ROCm refusal failed 4.

GREEN AFTER, on the repaired tree: test_cuda_fp8_kv_cache 7 cases / 10
assertions / SUCCESS!, exit 0, on a Release CPU build with -Wall -Wextra -Werror. It was 6 cases before; the extra case is G4b, and it skips here with
the other four device cases, each printing a MESSAGE naming what did not run.

No sibling regressions, same tree: test_ops_fp8_kv_cache 8 cases / 511
assertions, test_ops_reshape_cache 12 / 192, test_ops_paged_attn 14 / 1646,
test_ops_paged_attn_dtype 3 / 172, all SUCCESS!.

THE THREE CASES ADDED OR REWRITTEN HERE HAVE NO RED-FIRST MUTATION. G3's f16
leg, G4b and the rewritten G5 are device cases, this session has no nvcc and no
device for the same reason the implementing session had none, and that a CPU
build can neither compile nor execute them is measured above rather than assumed.
They are listed
under ## Owed in .agents/specs/fp8-kv-cache.md with the rest of the device
half, and the rc lease that runs ctest -R test_cuda_fp8_kv_cache owes their
mutations too.

Issue: #1593
Anchor debt filed: #1636

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]